A statistical mechanical interpretation of instantaneous codes

In this paper we develop a statistical mechanical interpretation of the noiseless source coding scheme based on an absolutely optimal instantaneous code. The notions in statistical mechanics such as statistical mechanical entropy, temperature, and thermal equilibrium are translated into the context of noiseless source coding. Especially, it is discovered that the temperature 1 corresponds to the average codeword length of an instantaneous code in this statistical mechanical interpretation of noiseless source coding scheme. This correspondence is also verified by the investigation using box-counting dimension. Using the notion of temperature and statistical mechanical arguments, some information-theoretic relations can be derived in the manner which appeals to intuition.Comment: 5 pages, Proceedings of the 2007 IEEE International Symposium on Information Theory, pp.1906 - 1910, Nice, France, June 24 - 29, 200

The Tsallis entropy and the Shannon entropy of a universal probability

We study the properties of Tsallis entropy and Shannon entropy from the point of view of algorithmic randomness. In algorithmic information theory, there are two equivalent ways to define the program-size complexity K(s) of a given finite binary string s. In the standard way, K(s) is defined as the length of the shortest input string for the universal self-delimiting Turing machine to output s. In the other way, the so-called universal probability m is introduced first, and then K(s) is defined as -log_2 m(s) without reference to the concept of program-size. In this paper, we investigate the properties of the Shannon entropy, the power sum, and the Tsallis entropy of a universal probability by means of the notion of program-size complexity. We determine the convergence or divergence of each of these three quantities, and evaluate its degree of randomness if it converges.Comment: 5 pages, to appear in the Proceedings of the 2008 IEEE International Symposium on Information Theory, Toronto, ON, Canada, July 6 - 11, 200

Properties of optimal prefix-free machines as instantaneous codes

The optimal prefix-free machine U is a universal decoding algorithm used to define the notion of program-size complexity H(s) for a finite binary string s. Since the set of all halting inputs for U is chosen to form a prefix-free set, the optimal prefix-free machine U can be regarded as an instantaneous code for noiseless source coding scheme. In this paper, we investigate the properties of optimal prefix-free machines as instantaneous codes. In particular, we investigate the properties of the set U^{-1}(s) of codewords associated with a symbol s. Namely, we investigate the number of codewords in U^{-1}(s) and the distribution of codewords in U^{-1}(s) for each symbol s, using the toolkit of algorithmic information theory.Comment: 5 pages, no figures, final manuscript to appear in the Proceedings of the 2010 IEEE Information Theory Workshop, Dublin, Ireland, August 30 - September 3, 201